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I.  INTRODUCTION 


A.  GENERAL 

The  increasing  complexity  of  our  every  day  jobs  requires  us  to  pursue  flexible  and 
adaptive  technologies  with  which  to  respond  to  our  professional  requirements.  One  such 
technology  is  an  expert  system,  often  referred  to  as  a  knowledge  based  system  (KBS). 
An  expert  system  is  essentially  a  computer  software  application  that  assists,  or  in  a 
number  of  situations  makes  decisions  based  on  the  problem  solving  capabilities  of  an 
expert.  It  actually  manipulates  knowledge.  [Ref.  l:p.  24]  This  computer  software  "tool" 
is  one  means  to  augment  and  streamline  one’s  professional  decision  making  process.  The 
expert  system  can  be  used  as  a  means  to  assist  new  or  inexperienced  people  to  make 
informed  decisions  about  their  jobs.  It  can  also  assist  in  the  decision  making  process 
when  the  technical  expert  is  not  present.  Due  to  the  fast  paced,  rapidly  changing  nature 
of  computer  software  development,  the  need  exists  for  a  specific  contracting  methodology 
for  use  in  the  development  and  acquisition  of  this  technology  within  the  Department  of 
Defense  (DoD). 

This  study  will  provide  an  objective  summary  and  initial  analysis  of  specific 
contractual  considerations  that  need  to  be  addressed  with  regard  to  the  acquisition  of  an 
expert  system. 
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B.  AREA  OF  RESEARCH  AND  OBJECTIVES 


This  research  effort  studies  the  varied  and  peculiar  contracting  issues  regarding  the 
acquisition  of  expert  systems.  For  the  purposes  of  this  research,  acquisition  refers  to  the 
procurement  of  the  total  system.  The  total  system  or  "life-cycle"  approach  encompasses 
the  entire  procurement  action  to  include  system  design,  specifications,  solicitation, 
contract  award,  and  system  maintenance.  The  object  of  this  research  is  to  review  current 
practices,  and  to  explore  options  available  to  program  managers  and  contracting  officers, 
that  may  improve  the  success  of  the  development  and  acquisition  of  expert  systems. 

C.  RESEARCH  QUESTIONS 

1.  Primary  Question 

What  unique  contractual  considerations  are  involved  in  procuring  an  expert 
system  for  use  in  the  DoD? 

2.  Subsidiary  Questions 

a.  Of  the  expert  systems  currently  acquired  by  the  DoD,  how  satisfied 
are  the  agencies  with  these  systems? 

b.  What  were  the  contractual  problems  associated  with  the  acquisition 
of  these  systems? 


c.  What  are  the  observations  of  industry  regarding  satisfaction  in  the 
application  and  performance  of  expert  systems,  and  any  special  considerations  involved 
in  the  acquisition  of  these  systems? 

d.  What  are  the  specific  issues  regarding  contract  type,  contract 
administration,  research  and  development  (R&D),  prototype  development,  production,  and 
maintenance  as  they  apply  to  the  acquisition  of  expert  systems? 

D.  SCOPE 

This  thesis  is  limited  to  the  survey  and  analysis  of  specific  issues  associated  with 
the  acquisition  of  expert  systems.  It  will  atten:q>t  to  identify  areas  peculiar  to  the 


2 


development,  acquisition,  and  contract  management  of  an  expert  system.  It  provides  an 
objective  summary  and  analysis  of  specific  contractual  considerations  that  need  to  be 
addressed  with  regard  to  the  development  and  acquisition  of  such  a  system. 

E.  METHODOLOGY 

The  research  methodology  used  for  this  study  included  the  topics  discussed  below. 

1.  Literature  Search 

A  literature  review  was  conducted  in  order  to  gain  familiarity  and  insight 
into  the  area  of  expert  systems,  and  to  determine  if  there  is,  in  fact,  a  need  to  identify  any 
peculiarities  involved  in  the  acquisition  of  such  a  system.  Initial  sources  of  information 
for  this  research  were  various  Federal  and  DoD  documents  addressing  the  acquisition  of 
automated  data  processing  equipment  and  software.  Additional  information  was  obtained 
from  the  Naval  Postgraduate  School  Library,  and  the  Defense  Logistics  Studies 
Information  Exchange.  The  review  consisted  of  both  Government  and  non-government 
publications. 

2.  Professional  Conference  Attendance 

Two  professional  conferences  were  attended  in  order  to  meet  personnel 
versed  in  the  area  of  expert  system  development,  procurement,  management,  engineering 
and  usage.  The  conferences  attended  are  listed  in  Appendix  A. 

3.  Interview 

Interviews  were  conducted  with  both  Government  and  non-govemment 
professionals.  The  interviews  were  obtained  via  personal  and  telephone  conversations 
among  people  with  varying  degrees  of  experience.  Expertise  in  the  area  of  expert  systems 
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ranged  from  program  managers  and  contracting  personnel,  to  software/system  engineers, 
to  research,  development,  test,  and  evaluation  personnel,  to  end  users.  The  interviews 
served  as  a  means  of  supplementing  and  augmenting  information  obtained  through  the 
literature  review  and  attendance  at  the  professional  conferences.  A  list  of  the  people 
interviewed  is  contained  in  Appendix  B. 

F.  LIMITATIONS 

Although  expert  systems  have  been  used  for  several  years,  there  is  virtually  no 
concise  documented  guidance  on  the  actual  acquisition  of  such  a  system.  While  there  is 
an  abundance  of  information  regarding  software  acquisition,  little  specifically  addresses 
expert  systems.  The  majority  of  information  has  been  through  interviews  and 
interpretation  of  actual  contracting  experiences  from  Government  and  non-Govemment 
sources.  Every  attempt  has  been  made  to  ensure  objectivity  and  focus  on  specific 
development  and  acquisition  contracting  issues  regarding  expert  system  acquisitions. 

G.  ORGANIZATION 

Chapter  n  presents  background  information  and  reviews  current  practices  regarding 
expert  systems  within  the  DoD.  Additionally,  the  level  of  user  satisfaction  with  these 
systems,  as  well  as  any  contractual  problems  associated  with  these  systems,  are  identified 
and  discussed. 

Chapter  III  addresses  current  practices  from  industry  regarding  expert  systems. 
The  level  of  satisfaction  with  these  systems,  along  with  the  contractual  methodologies  and 
problems  associated  with  these  systems,  are  identified  and  discussed. 
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Chjq)ter  FV  presents  the  varied  contract  types  available  as  outlined  in  the  Federal 
Acquisition  Regulation  (FAR).  A  discussion  of  the  importance  of  choosing  the 
appropriate  contract  type  for  the  situation  at  hand  is  addressed  in  terms  of  suitability  and 
need.  The  contracting  officer’s  role  in  this  situation  is  also  discussed. 

Chq)ter  V  draws  conclusions  from  the  information  gathered  in  this  research  effort. 
Recommendations  on  how  to  improve  and  enhance  the  procurement  of  expert  systems  for 
use  within  the  DoD  arena  are  proffered.  Finally,  the  (Thapter  closes  with  recommen¬ 
dations  for  future  research. 
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n.  BACKGROUND  AND  CURRENT  PRACTICES  IN  DoD 


A.  INTRODUCTION 

In  order  to  gain  the  appropriate  perspective,  it  is  relevant  to  note  how  this 
particular  research  topic  was  decided  upon.  Following  this  discussion,  and  prior  to 
addressing  current  practices  and  applications,  the  expert  system  will  be  defined,  and 
certain  identifying  characteristics  addressed. 

The  initial  focus  was  to  consider  the  contracting  requirements  of  an  expert  system 
titled  Expert  System  Advisor  for  Aircraft  Maintenance  Scheduling  (ESAAMS).  This 
project  is  in  the  early  stages  of  development  by  the  faculty  and  students  of  the  Naval 
Postgraduate  School.  Upon  initial  review,  it  became  obvious  that  little  research  had  been 
done  in  the  area  of  expert  system  acquisition.  For  this  reason,  this  research  effort  was 
altered  to  address  the  larger  arena  of  expert  system  acquisition  in  general  as  opposed  to 
a  specific  system.  The  broader  scope  of  the  research  will  be  useful  to  expert  system 
acquisitions,  vice  a  particular  application. 

One  way  to  define  an  expert  system  is  to  compare  it  with  an  ordinary  software 
program.  According  to  Waterman,  "the  most  basic  difference  is  tfiat  expert  systems 
manipulate  knowledge  while  conventional  programs  manipulate  data."  [Ref.  l:p.  24] 
Artificial  intelligence  (AI)  researchers  have  concluded  that  expert  systems  have  the 
following  distinguishing  characteristics: 

*  Expertise 

*  Symbolic  Reasoning 
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*  Depth 

*  Self-Knowledge 

An  expert  system  must  contain  "expertise".  It  must  "achieve  the  same  levels  of 
performance  in  the  domain  of  interest  that  human  experts  can  achieve."  [Ref.  l:p.  25] 
In  addition  to  expert  performance  and  skill,  the  system  must  have  breadth  or  "robusmess" 
as  well.  Waterman  maintains  that  this  area  is  one  of  the  least  developed  characteristics 
in  exp)ert  systems  today,  "but  one  that  hiunan  experts  can  do  easily."  [Ref.  l:p.  25] 

"Symbolic  reasoning"  means  rather  than  using  equations  or  algorithmic 
mathematical  computations  to  solve  problems,  an  expert  system  does  so  by  emphasizing 
the  choice  of  symbols.  "An  expert  system  manipulates  these  symbols  rather  than 
performing  standard  mathematical  computations."  [Ref.  l:p.  26] 

"Depth"  refers  to  its  effective  operating  capability  within  a  narrowly  defined  and 
challenging  domain.  Expert  systems  work  in  what  AI  scientists  call  "real-world  problem 
domains."  [Ref.  l:p.  26]  In  such  a  domain,  "the  problem  solver  q>plies  actual  data  to  a 
practical  problem  and  produces  solutions  that  are  useful  in  some  cost-effective  way."  [Ref. 
l:p.  27] 

"Self-knowledge"  or  "metaknowledge"  indicates  a  "knowledge  about  knowledge." 
[Ref.  l:p.  28]  This  is  an  inherently  important  characteristic  of  an  expert  system.  Such 
knowledge  allows  an  expert  system  to  understand  its  own  operation  in  addition  to 
containing  a  built  in  structure  that  facilitates  this  reasoning  process.  This  type  of 
knowledge  is  important  to  expert  systems  for  the  following  reasons: 

*  Users  tend  to  have  more  faith  in  the  results,  more  confidence  in  the 
system. 
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*  System  development  is  faster  since  the  system  is  easier  to  debug. 

*  The  assumptions  underlying  the  system’s  operation  are  made  explicit  rather 
than  being  implicit. 

*  It  is  easier  to  predict  and  test  the  effect  of  a  change  on  the  system 
operation.  [Ref.  l:p.  29] 

As  stated  previously,  the  purpose  of  this  research  effort  is  to  focus  on  the 
acquisition  aspects  of  an  expert  system  as  opposed  to  the  technical  aspects  of  the  system 
itself.  Given  the  above  definition  and  characteristics  of  an  expert  system,  the  following 
section  will  address  current  qjplications  within  the  DoD. 

B.  CURRENT  PRACTICES  IN  DoD 

Current  approaches  to  software  design  and  development  in  DoD  tend  to  be  one  of 
two  widely  used  methods.  The  classical  {q>proach  known  as  the  "waterfall  method"  lends 
itself  quite  well  to  current  Govenunent  practices.  The  waterfall  development  method 
requires  preliminary  definitions  of  requirements  and  detailed  speciHcations  or  statement 
of  work.  While  being  specifically  delineated,  the  rigid  design  specifications  often  serve 
to  hinder  or  reduce  the  flexibility  needed  by  the  contractor  in  developing  a  software 
system.  The  real  world  q)plication  of  the  waterfall  method  tends  to  greatly  inhibit  the 
flexibility  in  software  development.  The  restriction  stems  from  the  rigid  and  clearly 
defined  sequence  indicative  of  the  waterfall  model.  These  same  restrictions  hold  true  in 
contracting  for  an  expert  system  developed  using  a  waterfall  methodology.  [Ref.  2:pp.  34- 
35] 

Today,  the  generally  accepted  approach  to  contracting  for  a  software  system 
development  is  known  as  "rapid  prototyping."  This  is  an  iterative  process  which  allows 
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greater  user-developer  interface.  The  user  provides  initial  guidance  to  the  developer. 
These  specifications  are  often  ambiguous.  The  user  remains  in  constant  contact  with  the 
developer  throughout  the  system  development.  Initial  concepts  and  specifications  change. 
A  prototype,  however,  is  able  to  be  developed  rapidly  based  upon  initial  guidance  from 
the  user.  Because  of  the  user-developer  interface  throughout  the  process,  the  inevitable 
changes  that  occur  are  easier  to  respond  to  by  the  developer,  and  the  prototype  can  be 
changed  and  altered  as  necessary.  The  prototype  serves  as  a  working  example  for  the 
user  to  see  if  the  system  really  works  as  intended.  Multiple  prototype  iterations  are  made, 
each  expanding  the  scope  of  the  system.  The  close  coordination  and  commimication 
between  the  ultimate  end-user  and  the  developer  ensure  that  the  final  system  is  in  fact 
what  the  user  intended.  [Ref.  2:pp.  35-37] 

While  rapid  prototyping  seems  an  efficient  means  of  developing  a  software  system, 
it  is  not  necessarily  in  line  with  current  Government  acquisition  methodologies.  These 
methodologies  are  predominantly  hardware  oriented.  Present  regulations  require  that 
specifications  be  delineated  in  the  contract  prior  to  contract  award.  The  rqjid  prototyping 
method  calls  for  initial  "broad  functional  descriptions  to  loosely  define  the  end  product’s 
objectives"  vice  the  work  to  be  performed.  [Ref.  2;p.  37]  Given  the  disparity  between 
what  method  is  most  efficient,  and  what  the  system  calls  for.  Government  acquisition 
personnel  are  attempting  to  fit  the  rapid  prototyping  method  into  the  present  hardware 
system.  At  the  same  time,  they  are  looking  at  ways  to  make  the  development  process  for 
expert  systems  adapt  to  the  peculiar  needs  of  conventional  software  development  and 
acquisition. 
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So  far,  this  discussion  has  focused  on  the  methodologies  used  for  developing  and 
fielding  computer  software  programs  in  the  Federal  Government.  Because  expert  systems 
are  software,  their  development  and  acquisition  would  seem  to  be  of  a  similar  nature. 
Although  there  are  similarities  between  the  accepted  methods  for  procuring  expert  systems 
and  common  software  applications,  there  are  also  differences.  Before  addressing  the 
differences,  however,  it  is  necessary  to  note  a  few  of  the  recognized  myths  and  facts 
regarding  expert  systems  projects. 

Mr.  A.F.  Umar  Khan,  Program  Analysis  Division,  7th  Communications  Group 
(United  States  Air  Force),  has  done  extensive  research  into  streamlining  the  development 
and  acquisition  of  expert  system  applications.  His  findings  and  current  DoD  trends  in  the 
development  and  acquisition  of  expert  systems  have  been  noted  in  several  pjq)ers  and 
presentations.  At  an  Institute  of  Electrical  and  Electronics  Engineers,  Inc.  (IEEE) 
conference  on  "Managing  Expert  System  Programs  and  Projects"  held  on  September  10- 
12,  1990  in  Bethesda,  Maryland,  he  addressed  his  current  Endings.  He  has  identified 
myths  and  facts  regarding  expert  system  development  and  acquisition.  They  are  listed  in 
Table  2-1. 

Given  these  myths  and  facts  about  expert  systems,  it  becomes  evident  that  expert 
systems  development  and  acquisition  techniques  are  in  fact  similar  to  conventional 
software  methodologies.  Mr.  Khan  noted  that  "a  recurring  point"  in  his  research  was  that 
expert  system  development  and  acquisition  projects  are  essentially  software  engineering 
efforts.  By  viewing  these  projects  as  software  engineering  efforts,  "lessons  learned 
building  software  over  the  past  decades  still  apply."  [Ref.  4:p.  33]  He  also  stressed  that 
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TABLE  2-1 


MYTHS  AND  FACTS 
Source:  Ref.  3:  3 


MYTHS: 

* 

Expert  system  development  should  not  be  managed  by  the 
traditional  management  information  system  (MIS)  department. 

* 

Expert  system  development  is  radically  different  from 
conventional  software  development,  i.e.,  existing  standards 
cannot  be  applied. 

* 

Expert  system  development  varies  too  much  and  is  dependent 
on  the  application,  i.e.,  it  is  impossible  to  establish  a  single 
model. 

FACTS: 

* 

Expert  system  development  projects  are  software  engineering 
efforts,  i.e.,  lessons  learned  building  software  over  the  past  de¬ 
cades  can  be  applied. 

* 

Most  peculiarities  of  expert  system  development  which  have 
been  much  publicized  in  the  popular  literature  are  not  totally 
unique  to  expert  system  projects. 

* 

Failure  to  provide  familiar,  comfortable  management  controls 
contributes  to  low  acceptance  of  expert  systems. 

these  "lessons  learned"  can  also  be  applied  to  expert  systems.  He  indicated  that  expert 
systems  should  be  viewed  in  a  similar  context  as  conventional  software. 

Prior  to  addressing  the  recommended  methodologies  for  acquiring  an  expert 
system,  it  should  be  noted  that  there  are  essentially  two  ways  to  acquire  a  system 
regardless  of  system  size.  One  method  is  to  acquire  a  standard  commercial  "off  the  shelf 
program  "shell".  Depending  upon  the  application,  funding  constraints,  and  time 
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requirements,  the  requiring  organization  may  opt  to  purchase  a  ready  made  commercial 
expert  system  "shell".  ITie  "shell"  allows  the  requiring  organization  to  tailor  the  expert 
system  to  their  specific  organizational  needs.  It  provides  the  problem  solving  capabilities 
of  an  expert  system  at  less  cost  than  if  the  organization  sought  to  develop  its  own  system. 

Another  option  is  to  program  the  expert  system  from  scratch.  In  this  instance, 
additional  considerations  need  be  addressed.  Along  with  cost,  application,  time  and 
funding  constraints,  the  requiring  activity  wdl  have  to  consider  the  availability  of  trained 
MIS  personnel,  resident  "experts",  a  focused  and  specialized  application,  and  maintenance 
over  the  life  of  the  system. 

Mr.  Khan’s  study  primarily  addresses  medium  to  large  scale  expert  system 

projects,  and  proffers  a  life-cycle  approach  to  the  development  and  acquisition  process. 

His  recommended  approach  attempts  to  merge  expert  system  development  and  acquisition 

to  the  DoD  life-cycle  model.  [Ref.  5:p.  16]  He  identifies  the  importance  of  this  approach 

in  that  these  systems  "require  more  documentation  and  control  dining  development  than 

the  smaller,  well  defined,  highly  constrained,  stand-alone  systems."  [Ref.  4;p.  32]  He 

adds,  however,  that  "it  is  preferable  that  small  systems  comply  with  the  same  life-cycle 

model  as  larger  systems  and  that  they  adhere  to  similar  control  processes."  [Ref.  4:p.  32] 

An  explanation  of  the  recommended  life-cycle  approach  will  be  given  later  in  the  Chapter. 

C.  COMMONLY  CITED  PROBLEM  AREAS  IN  EXPERT  SYSTEM  DEVEL¬ 
OPMENT  AND  ACQUISITION 

Throughout  this  research  effort,  there  have  been  recurring  references  to  certain 
areas  of  concern.  Both  in  personal  interviews  and  conference  presentations,  similar 
concerns  have  been  raised  regarding  problems  associated  with  the  acquisition  and 
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development  of  expert  systems.  Similarly  related  concerns  have  been  addressed  in  the 
p{q)ers,  reports,  and  books  reviewed  in  conjunction  with  this  research.  The  following 
discussion  presents  the  commonly  noted  areas  of  concern  from  the  Federal  Government 
perspective.  The  interview  techniques  used  in  obtaining  this  information  were  face  to 
face  and  telephone  type  interviews.  A  discussion  of  industry’s  concerns  and  conclusions 
follows  in  the  next  Chapter. 

1.  Five  Most  Cited  Problems  Regarding  Expert  System  Acquisition  and 

Development  In  DoD 

A  study  on  software  contracting  conducted  by  Major  Henry  Attanasio  in 
1990  was  used  as  a  reference  in  helping  interviewees  to  identify  their  concerns.  [Ref.  2: 
p.  49]  The  five  most  commonly  cited  problems  for  expert  systems  development  from  the 
DoD  personnel  perspective  are  listed  in  Table  2-2. 

2.  Recommended  Rationale  And  Possible  Solutions  To  The  Problems 

Based  on  the  interviews,  two  major  issues  were  raised  in  linking  the  cited 

problems  with  a  cause  and  potential  solution.  The  two  issues  are: 

*  Development  and  acquisition  methodology 

*  Training  qualified  and  experienced  personnel 

As  previously  discussed,  the  currently  preferred  method  of  rapid 
prototyping  is  one  method  in  which  currently  perceived  problems  can  be  addressed.  By 
using  the  r£q>id  prototyping  methodology,  requirements  issues  are  responded  to  as  part  of 
the  development  process.  Additionally,  concerns  regarding  reliability  and  maintainability 
are  addressed  as  a  normal  part  of  the  development  process.  The  interface  between  the 
user  and  contractor  throughout  the  process  ensures  these  issues  are  addressed.  [Ref.  6] 
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TABLE  2-2 


COMMONLY  CrTED  PROBLEMS  FROM  DOD 
Source:  Ref.  2:  49 


Lack  of  qualified  Government  technical  personnel 
Unclear  user  requirements 

Lack  of  understanding  regarding  expert  system  design  and 
development 

Difficulty  in  measuring  reliability  and  maintainability  of  expert 
systems 

Too  many  changes  to  initial  requirements 


In  order  for  the  rapid  prototyping  method  to  work,  ttiere  exists  a  need  for 
qualified  and  trained  people.  The  user  has  to  be  familiar  with  software  terminology  and 
the  technical  aspects  of  computer  systems  {qrplications,  as  well  as  willing  to  learn  about 
expert  systems.  The  user  should  not  only  be  well  versed  in  determining  the  specifications 
or  statement  of  work,  but  must  decide  on  who  will  maintain  the  system.  The  interviewees 
agreed  that  the  more  knowledgeable  the  user  is,  the  easier  the  development,  acquisition, 
and  implementation  will  be. 

D.  CONTRACTING  ISSUES  REGARDING  EXPERT  SYSTEMS 

Specffically  addressing  expert  systems  acquisition  and  development  in  terms  of 
contracting  issues  is  of  paramount  importance.  The  contract  is  perhaps  the  single  most 
important  document  in  the  acquisition  process.  It  addresses  the  issues  of  cost,  schedule. 
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technical  performance,  and  maintenance  support,  not  to  mention  the  question  of  risk,  legal 
requirements,  and  administration.  The  contract  serves  as  the  guiding  document  by  which 
the  entire  acquisition  is  conducted.  It  is  not  just  a  legal  basis  for  performance,  but  a 
vehicle  through  which  responsibilities  are  delineated  for  both  the  contractor  and  the 
Government.  This  section  will  address  the  issues  stated  above.  The  contract  type,  which 
plays  an  imponant  role  in  assigning  the  level  of  risk  to  either  party,  is  discussed  in 
Chq>ter  IV. 

1.  Cost 

Cost  plays  an  important  role  in  that  it  determines  what  the  total  price  of 
the  system  will  be.  Price  is  the  sum  of  cost  plus  a  fair  and  reasonable  proHt.  It  is  easy 
to  see  the  importance  of  cost  because  profit  is  largely  based  on  it. 

For  expen  systems,  cost  can  be  a  difficult  issue  to  understand.  Cihan  H. 
Dagli  of  the  University  of  Missouri-Rolla,  identified  two  kinds  of  expert  system  costs: 
"One  Time"  and  "Ongoing".  The  following  "one  time"  costs  were  identified: 

*  Software  shell  purchase 

*  Software  development 

*  Other  software  purchase 

*  Hardware  lease  or  purchase 

*  Communication  equipment 

*  Office  space  and  fomishings 

*  Training  and  documentation 

In  addition  to  the  "one  time"  costs,  the  following  "ongoing"  or  "recurring" 
costs  were  also  identified: 

*  Operating  personnel 

*  Communication  lines 

*  Hardware  maintenance 
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*  Software  upgrades 

*  Office  space  and  utilities  [Ref.  7:p.  119] 

While  not  specifically  stated  in  the  above  terms,  most  of  the  DoD  people 
interviewed  were  aware  of  the  above  costs.  The  key  areas  that  DoD  people  seem  to  focus 
on  regarding  cost  of  a  system  were  software  development,  training  and  education, 
operating  personnel,  and  software  upgrades.  Regardless  of  the  costs,  and  the  fact  that  AI 
technology  is  a  new  and  rapidly  changing  field,  an  understanding  of  costs  associated  with 
expert  systems  acquisition  by  contracting  personnel  can  help  reduce  these  costs  to  the 
Government. 

By  understanding  the  costs  associated  with  expert  system  development  and 
acquisition,  the  Government  can  realistically  deal  with  contractors.  According  to  Mr. 
Khan,  by  following  the  iterative  rapid  prototyping  £q>proach  for  development  and 
acquisition,  many  costs  can  be  significantly  reduced.  Inherent  in  the  rapid  prototyping 
methodology  are  management  controls.  Such  controls  govern  the  user-developer  interface 
in  addition  to  any  design,  development,  and  change  actions.  Because  of  the  degree  of 
user  involvement  throughout  the  process,  the  user  will  likely  be  satisfied  with  the  end 
product  at  final  delivery.  The  likelihood  of  a  large  number  of  changes  and  modifications 
after  final  delivery  is  lower  in  this  process  as  opposed  to  one  which  requires  rigid 
specifications  up-front.  [Ref.  6] 

2.  Schedule 

The  primary  discussion  of  schedule  as  a  part  of  the  contract  terms  was  in 
relation  to  the  rapid  prototyping  methodology  for  expert  system  development  and 
acquisition.  Rapid  prototyping  combined  with  the  life  cycle  approach  recommended  by 
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Mr.  Khan  seemed  to  be  the  widely  accepted  vehicle  for  controlling  the  development  as 
well  as  deliveiy  schedule. 

Mr.  Khan  noted  the  varying  degrees  of  schedule  risk  associated  with 
different  stages  of  the  system  development  process.  The  degree  of  schedule  risk  tends 
to  decrease  as  the  process  changes  from  concept  development,  to  design,  to  prototype 
development.  As  will  be  discussed  in  Chapter  IV,  when  combined  with  an  incentive  type 
contract,  the  Government  tends  to  have  greater  control  over  the  contraaor.  Similarly,  the 
contractor  is  incentivized  to  deliver  on  time.  [Ref.  6] 

3.  Technical  Performance 

Performance  of  the  system,  as  described  in  the  contract  is  obviously  a  key 
ingredient  of  the  overall  procurement.  While  the  specific  performance  requirements  will 
vary  depending  on  the  system  usage,  the  unique  performance  requirements  for  a  specific 
application  should  be  specifred  in  the  contract.  In  accordance  with  the  currently  accepted 
rapid  prototyping  methodology,  preliminary  performance  is  based  upon  "broad  functional 
descriptions  that  loosely  define  the  end  product’s  objectives."  [Ref.  2:p.  37]  By  allowing 
the  contractor  a  certain  degree  of  latitude  in  developing  an  expert  system,  the  system 
prototype  can  be  viewed  as  a  test  bed  for  the  user.  The  prototype  allows  the  user  to 
actually  see  if  the  system  will  meet  given  requirements.  The  prototype  also  allows  the 
user  to  specify  further  requirements,  and  the  contractor  to  implement  and  respond  to 
suggested  changes. 

Mr.  Jay  Griesser,  Technical  Application  Branch,  Navy  Finance  Center, 
Cleveland,  Ohio,  specifically  addressed  the  issue  of  technical  performance.  While  the 
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tinance  center  has  conducted  its  expert  system  development  "in  house",  Mr.  Griesser 
stressed  the  importance  of  technical  performance.  He  indicated  that  the  users  were  able 
to  remain  in  continuous  contact  with  the  knowledge  engineers  and  system  developers 
throughout  the  development  process.  In  a  similar  vein  as  the  user-contractor  relationship 
in  the  rapid  prototyping  methodology,  the  ability  for  the  user  to  evaluate  the  system  from 
a  technical  performance  aspect  is  not  only  necessary,  but  "should  be  required."  [Ref.  8] 
This  system  evaluation  by  the  user  is  inherent  within  the  r^id  prototyping  process.  By 
requiring  the  evaluation  as  part  of  the  terms  and  conditions  of  the  contract,  and  effective 
management  control  by  the  program  manager  and  contracting  officer,  technical 
performance  can  be  reasonably  assured. 

4.  Maintenance  Of  Expert  Systems 

While  largely  ignored  in  literature  as  well  as  professional  conferences,  and 
paper  presentations,  the  issue  of  expert  system  maintenance  is  important.  Maintaining  the 
system,  or  undertaking  "a  set  of  software  engineering  activities  that  occur  after  software 
has  been  delivered  to  the  customer  and  put  into  operation",  is  one  way  of  looking  at  the 
maintenance  issue.  [Ref.  9:p.  1]  Current  estimates  indicate  software  maintenance  entails 
as  much  as  seventy  percent  of  the  overall  software  life-cycle  costs  for  any  given  software 
project.  [Ref.  9:p.  1]  Knowledge  in  an  expert  system  must  be  current  if  the  system  is  to 
be  used.  It  will  require  constant  updating.  The  potential  for  expert  system  maintenance 
costs  to  be  as  great  or  even  greater  than  software  development  expenses  is  real. 

One  way  of  reducing  or  controlling  maintenance  costs  was  to  "do  all  of  the 
maintaining  in  house."  [Ref  8]  In  doing  so,  an  activity  could  save  on  the  costs  of 
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contracting  out  for  maintenance  support.  Due  to  die  nature  and  complexity  of  the  varied 
s^lications  of  expert  systems,  the  interviews  indicated  that  in-house  maintenance  was  the 
exception  and  not  the  norm.  Training  and  education  for  experienced  and  qualified 
personnel  will  be  required  to  conduct  in-house  maintenance.  [Ref.  8] 

A  final  problem  stemming  from  the  issue  of  expert  systems  maintenance 
is  cost.  Not  the  cost  of  contracting  for  the  required  maintenance,  but  the  cost  incurred 
in  the  development  and  implementation  stage  of  the  acquisition  process.  In  considering 
this  issue.  Professor  Martin  J.  McCaffrey,  Naval  Postgraduate  School,  Monterey, 
California,  suggests  that  because  of  the  "significant  costs  of  future  maintenance,  these 
issues  are  often  ignored  in  development  and  in^lementation."  Program  managers  or 
contracting  officers  may  not  be  adequately  addressing  this  issue.  [Ref.  9:p.  S]  Prof. 
McCaffrey  states  that  failure  to  address  maintenance  issues  during  development  is  a 
"significant  impediment  to  both  the  growth  of  expert  systems  in  the  future  and  the  life 
cycle  management  of  these  systems."  [Ref.  9:p.  8] 

E.  EXPERT  SYSTEM  DEVELOPMENT  PROCESS  AND  THE  DoD  LIFE- 
CYCLE  MODEL 

The  following  section  presents  the  DoD  life-cycle  process  for  automated 
information  systems  (AIS)  as  outlined  in  DoD  Directive  7920.1.  The  expert  systems  (ES) 
development  process  is  then  "mapped"  onto  the  DoD  model.  [Ref.  5:p.  16] 

The  various  phases  as  outlined  in  the  DoD  model  for  AIS  development  are 
identified  in  Table  2-3.  These  phases  are  for  the  development  of  conventional  software 
systems,  and  are  not  peculiar  to  the  development  of  expert  systems. 
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TABLE  2-3 


DOD  MODEL  FOR  AIS  DEVELOPMENT 
Source:  Ref.  5:  16 


Needs-Justification  Phase 

Milestone  0  (Decide  WHAT  is  wanted.) 
Concepts-Development  Phase 

Milestone  I  (Decide  HOW  to  do  it.) 

Design  Phase 

Milestone  n  (Decide  if  DESIGN  is  OK.) 
Development  Phase 

Milestone  in  (Decide  if  SYSTEM  is  OK.) 
Deployment  Phase 

Milestone  IV  (Decide  if  successful.) 

Operations  Phase 

Milestone  V  (Decide  if  new  system  is  needed.) 


The  development  process  for  expert  systems,  as  identified  by  Mr.  Khan,  is 
portrayed  in  Table  2-4.  This  process  is  based  on  the  rapid  prototyping  methodology,  and 
applies  specifically  to  expert  systems.  It  addresses  the  expert  system  development  from 
initial  design,  through  the  prototype  iterations,  and  on  to  actual  deployment. 

Table  2-5  portrays  the  DoD  model  for  AIS  development  and  the  proposed  ES 
development  model  "mapped"  together. 
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TABLE  2-4 


PROPOSED  ES  DEVELOPMENT  PROCESS 
Source:  Ref.  5:  16 


Initiation  Phase 
Concept  Prototype 
Demonstration  Prototype 
Testbed  Prototype 
Operational  Prototype 
Deployment  Phase 
Post-Deployment  Phase 


ES  PROCESS 

"MAPPED”  ONTO  THE  DOD  MODEL 

Source:  Ref.  5:  16 

AIS 

ES 

Needs-Justification  Phase 

Initiation  Phase 

Milestone  0 

Concepts-Development  Phase 

Concept  Prototype 

Milestone  I 

Design  Phase 

Milestone  n 

Demonstration  Prototype 

Testbed  Prototype 

Development  Phase 

Milestone  IQ 

Operational  Prototype 

Deployment  Phase 

Milestone  IV 

Deployment  Phase 

Operations  Phase 

Milestone  V 

Post-Deployment  Phase 
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The  basic  premise  of  "fitting"  the  r^id  prototyping  model  to  the  DoD  model  is 


to: 

Woik  with  the  model  currently  followed  throughout  DoD  for  AIS 
procurement. 

♦  Ensure  that  rq)id  prototyping  is  the  method  of  choice  for  expert  system 
development  and  acquisition. 

Although  expert  systems  to  date  tend  not  to  be  as  large  as  the  systems  addressed  in  the 
DoD  model,  the  expert  systems  model  "can  be  mq>ped  straightforwardly  into  the  AIS  life- 
cycle  management  phases  of  7920.1."  [Ref.  5:p.  15]  By  following  this  recommended 
process,  issues  of  integration,  user  particq}ation,  and  dynamic  documentation  are  more 
adequately  addressed  in  the  realm  of  the  total  or  life-cycle  {q>proach  to  expert  systems 
development  and  acquisition. 

F.  SUMMARY 

This  Chapter  has  presented  an  overview  of  the  background,  current  practices  and 
applications,  and  specific  contracting  related  issues  regarding  expert  system  development 
and  acquisition.  While  not  addressing  specific  expert  systems,  this  Chapter  focused  on 
commonly  cited  problems  and  contracting  issues  deemed  significant  to  understanding 
current  trends.  By  identifying  the  problems,  as  well  as  presenting  pertinent  contracting 
issues,  the  researcher  is  attempting  to  set  the  stage  for  presenting  various  conclusions  and 
recommendations  in  Chapter  V. 
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m.  CURRENT  PRACTICES  IN  INDUSTRY 


A.  INTRODUCTION 

In  order  to  gain  an  accurate  picture  of  current  endeavors  in  the  acquisition  of 
expert  systems,  it  is  important  to  contrast  the  efforts  of  the  Government  with  a  look  at 
industry.  The  previous  Chapter  discussed  the  concerns  from  DoD  representatives.  This 
Chapter  will  present  areas  of  concern  from  industry  as  well  as  particular  contraaing 
issues  deemed  important  by  industry  professionals.  Interviews  were  the  primary  source 
of  findings. 

B.  COMMONLY  CITED  PROBLEM  AREAS  IN  EXPERT  SYSTEM  DEVEL¬ 
OPMENT  AND  ACQUISITION 

Similar  to  their  DoD  colleagues,  industry  professionals  were  specific  and  direct 
in  determining  and  identifying  problems  associated  with  the  development  and  acquisition 
of  expert  systems.  The  majority  of  comments  solicited  from  industry  representatives  dealt 
with  expert  systems  that  were  designed,  developed,  and  acquired  for  a  specific  application 
as  opposed  to  commercial  "off-the-shelf  shells. 

1.  Five  Most  Commonly  Cited  Problems  From  Industry  Regarding 
Expert  Systems  Acquisition 

The  five  most  commonly  cited  problems  from  industry  regarding  expert 
systems  acquisition  are  identified  in  Table  3-1.  As  with  the  Government  personnel 
interviewed.  Major  Attanasio’s  list  of  problems  was  used  as  an  aid. 
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TABLE  3-1 


COMMONLY  CITED  PROBLEMS  FROM  INDUSTRY 
Source:  Ref.  2:  49 


Unclear  user  requirements 
Too  many  changes  to  initial  requirements 
Too  many  Government  regulations 
Too  many  Government  specifications 
Inadequate  Government  specifications 


2.  Recommended  Rationale  and  Possible  Solutions  To  The  Problems 

While  somewhat  different  from  the  problems  identified  in  the  Government 
perspective,  there  were  two  striking  similarities.  The  issues  of  unclear  user  requirements, 
and  the  number  of  changes  to  the  initially  stated  requirements,  q>peared  as  significant 
areas  of  concern  in  both  the  public  and  private  sector.  These  two  problems  were  linked 
together  because  preliminary  user  requirements  are  often  unclear.  Changes  are  required 
further  on  in  the  expert  system  development  and  acquisition  process. 

While  the  link  between  the  two  problems  seem  evident,  the  recommended 
solutions  are  not.  Industry  professionals  feel  that  better  training  and  education  on  the  part 
of  the  ultimate  "end-user"  would  aid  users  in  identifying  requirements.  In  a  similar  vein, 
industry  feels  that  closer  coordination  with  the  user  is  essential  to  developing  a  system 
that  will  be  "everything  the  user  intended."  [Ref.  10]  By  using  the  rapid  prototyping 
methodology,  industry  representatives  feel  that  the  "necessary"  level  of  user-developer 
"interface"  would  be  ensured. 
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The  remaining  commonly  identified  problems  refer  to  various  issues 
regarding  specifications  and  regulations.  Most  people  interviewed  understood  diat  the 
Federal  Government  was  involved  in  steps  to  streamline  the  acquisition  process.  Taking 
a  hard  look  at  the  "over-abundance"  of  regulations  is  viewed  as  a  step  in  the  right 
direction.  Although  the  issue  of  Government  regulations  was  identified  as  "problematic" 
and  "cumbersome",  no  specific  regulations  were  identiHed. 

The  issue  of  specifications  is  the  area  industry  people  emphasized  as  being 
most  problematic.  Although  not  an  industry  representative,  Mr.  Khan,  who  regularly 
deals  with  industry  professionals,  indicated  that  identifying  and  reviewing  specifications 
is  where  the  majority  of  time  and  effort  should  be  spent.  He  emphasized  the  importance 
of  including  such  information  as  the  purpose  and  scope  of  the  system,  as  well  as  a  de- 
scr^tion  of  the  development  and  delivery  environment.  [Ref.  6] 

Specifications,  as  described  by  Major  Attanasio  "are  complete  and  detailed  descrip¬ 
tions  of  products  which  are  cither  military  in  nature,  or  are  modified  commercial  products 
requiring  special  features  to  satisfy  military  mission  needs."  [Ref.  2:pp.  18-19]  Given  this 
description,  industry  people  felt  diat  the  technical  specifications  tend  to  be  problematic. 
It  is  this  area  where  developers  tend  to  have  a  lot  of  difficulty  due  to  specifications  not 
being  complete  or  detailed  enough.  On  the  other  end  of  the  spectrum,  contractors  tended 
to  feel  that  specifications  were  sometimes  too  restrictive  in  nature  thereby  limiting  the 
development  process,  although  no  specific  examples  were  cited. 

The  question  of  adequate  training  and  familiarity  on  the  part  of  the  Government 
in  determining  system  specifications  seemed  to  be  a  recurring  theme.  Mr.  Jim  Crossen, 
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of  TRW,  Sunnyvale,  California,  stressed  the  in^rtance  of  adequate  education  by 
Govenunent  people  associated  with  the  procurement  of  expert  systems  and  software  in 
general.  Such  education  should  include  familiarity  with  basic  computer  hardware  and 
software  terminology.  In  addition  to  being  familiar  with  the  terminology,  procurement 
personnel  must  be  able  to  envision  or  conceptualize  what  it  is  they  are  dealing  with. 
While  the  discussion  with  Mr.  Crossen  addressed  contract  types  and  contracting  issues, 
he  did  indicate  a  concern  that  contracting  officers  "get  up  to  speed"  when  dealing  with 
highly  technical  procurements  such  as  expert  systems.  [Ref.  11]  In  other  words,  they 
need  to  understand  what  exactly  it  is  that  they  are  contracting  for.  Through  both  formal 
and  informal  education,  the  requisite  level  of  understanding  and  familiarity  can  be 
attained. 

In  a  similar  vein,  Mr.  Conner,  of  InteUicorp,  mentioned  not  only  the  importance 
of  adequate  technical  familiarity,  but  also  a  concern  over  the  number  of  Government 
specifications  and  standards.  [Ref.  10]  While  not  being  specific,  he  did  indicate  that  a 
number  of  regulations  and  standards  for  the  management  of  DoD  software  projects  are 
redundant.  A  review  of  the  standards  and  regulations  identified  by  Major  Attanasio 
seemed  to  reflect  this  point.  For  example,  even  though  DoD-STD-2167A  addresses  "all 
aspects  of  software  design,  development  and  testing",  DoD  STD- 1467  and  DoD-STD- 
1703  address  software  acquisition  from  a  program  manager’s  and  life  cycle  perspective. 
[Ref.  2;pp.  19-21]  While  being  important  topics  to  be  looked  at,  there  exists  certain 
degrees  of  redundancy  and  overlap  between  these  standards.  A  review  of  the  Government 
standards  and  handbooks  presented  by  Major  Attanasio  indicates  that  industry’s  feeling 
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regarding  Government  regulations,  speciHcations,  and  standards  has  some  merit.  Both 
Government  and  industry  tend  to  feel  that  these  areas  should  be  looked  at  more  closely. 

Industry  representatives  felt  that  streamlining  the  Government  acquisition  process, 
as  well  as  enqrhasizing  rapid  prototyping  for  expert  systems  and  software  in  general,  is 
the  right  direction  to  be  heading.  By  not  only  streamlining  the  acquisition  organization, 
but  the  rules  and  regulations  governing  the  system,  a  lot  of  time  and  effort  can  be  saved 
in  the  Government  as  well  as  industry.  Mr.  Crossen  directly  related  this  streamlining  to 
savings  in  "cost,  schedule,  and  management"  areas.  [Ref.  11] 

C.  CONTRACTING  ISSUES  REGARDING  EXPERT  SYSTEMS 

Of  the  industry  professionals  interviewed,  Mr  Crossen  was  by  far  the  most 
outspoken  representative.  As  indicated  in  the  previous  section,  he  specifically  stressed 
the  importance  of  "cost,  schedule,  and  management  control."  [Ref.  11]  For  this  reason, 
and  in  keeping  with  the  format  followed  in  the  previous  Chapter,  the  contracting  issues 
regarding  expert  systems  will  be  broken  down  into  the  areas  of  cost,  schedule,  technical 
performance,  and  maintenance  support.  Management  control  will  be  linked  directly  to 
each  of  these  areas  throughout  the  discussion. 

1.  Cost 

Cost  is  an  important  issue  in  any  contract.  In  the  area  of  expert  systems 
development  and  acquisition,  it  is  just  as  important.  Based  upon  the  "one-time"  and 
"recurring  "  costs  identified  in  the  previous  Chapter,  industry  representatives  viewed  these 
as  being  accurate  and  realistic.  The  industry  perspective  differed  somewhat  from  the 
Government.  The  Government  placed  emphasis  on  costs  associated  with  software 


27 


development,  training  and  education,  operating  personnel,  and  software  upgrades.  The 
industry  perspective,  however,  tended  toward  software  development,  maintenance, 
acquisition  method,  and  contract  type.  The  maintenance  issue  will  be  addressed  in  the 
appropriate  section. 

While  closely  related,  software  development  and  acquisition  methods  were 
generally  discussed  as  two  separate  issues,  although  a  link  was  always  drawn  between  the 
two. 

2.  Schedule 

The  delivery  schedule  is  widely  recognized  as  a  significant  part  of  the 
contracting  process.  Largely  linked  with  the  development  method,  industry  representa¬ 
tives  indicated  that  during  the  development  phase,  it  is  important  to  concentrate  on 
"common  sense  and  rationality."  [Ref.  11]  In  later  stages  of  the  system  development,  a 
concise  and  restrictive  schedule  can  often  be  agreed  to  with  the  assurance  that  delivery 
will  be  on  time.  Industry  professionals  seemed  to  be  unanimous  in  the  belief  that  rapid 
prototyping  combined  with  the  DoD  life  cycle  q>proach  as  presented  by  Mr.  Khan  is  the 
preferred  method. 

Similar  feelings  regarding  the  link  between  schedule  and  the  development 
methodology  were  discovered  between  industry  representatives  and  DoD  people.  This 
sense  of  agreement  between  the  private  and  public  sectors  was  not  surprising  in  that  the 
two  conferences  which  were  attended  as  part  of  this  research  consisted  of  DoD  and 
industry  representatives  alike. 
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3. 


Technical  Performance 


Industry  representatives  seemed  to  stress  the  importance  of  technical 
performance.  Technical  performance  refers  to  not  only  the  physical  performance  of  the 
system,  but  the  interface  with  the  user  and  maintenance  technicians.  This  interface  is  a 
direct  result  of  how  well  the  system  meets  the  specifications  requirements  as  identified 
in  the  contract.  The  performance  of  the  proposed  system  was  viewed  as  an  essential 
concern  for  both  parties  involved.  Because  of  the  rapidly  growing  technology  in  the  AI 
field,  industry  people  seemed  to  stress  the  current  technological  advances  in  this  area. 
Again,  the  iterative  process  of  developing  expert  systems,  or  software  in  general,  lends 
itself  to  being  vulnerable  to  technology  change  at  every  phase  of  design,  development  and 
production.  Because  of  this  vulnerability,  it  is  important  to  note  that  depending  upon  the 
intended  application  of  the  system,  the  latest  technology  may  not  necessarily  be  the  best 
alternative. 

Mr.  Francisco  J.  Cantu-Ortiz,  of  the  Institute  Tecnologico  y  de  Estudios 
Superiores  de  Monterrey,  Monterrey,  Mexico,  specifically  addressed  the  in^rtance  of 
technology  transfer  from  the  university  to  industry.  At  the  Managing  Expert  Systems 
(MES)  90  conference,  he  proposed  several  strategic  goals  for  the  furtherance  of  AI 
technology.  Among  his  list  of  proposed  strategies  are:  applied  research  and  development, 
an  aimual  program  of  seminars  and  symposiums  on  expert  systems,  and  research 
agreements  with  industry.  [Ref.  12:p.  70]  These  suggested  strategies  stress  the  importance 
of  technology.  These  and  similar  efforts  in  the  U.S.  indicate  the  rapidly  changing  pace 
of  expert  systems  technology.  Because  of  the  changing  environment,  it  becomes  even 
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more  important  for  the  Government  and  industry  to  maintain  a  close  working  relationship 
or  interface  throughout  the  development  process.  This  relationship  thereby  ensures 
acceptable  technical  performance  of  the  final  product  through  the  iterative  development 
process. 

4.  Maintenance  Support 

When  discussing  the  aspects  of  maintenance  support,  industry 
representatives  viewed  this  as  a  concern,  but  never  really  addressed  tire  issue  in  detail. 
Most  agreed  with  the  need  for  some  kind  of  post  delivery  maintenance  such  as 
consultants,  knowledge  engineers,  in-house  maintenance  and/or  contracted-out  type 
maintenance  support,  however,  no  specific  reconunendations  were  made.  In  acknowledg¬ 
ing  maintenance  support  problems  like  documentation,  user  requests  for  changes,  and 
quality  of  the  software,  Mr.  Ed  Hoffinan  of  Intellicorp  suggested  that  it  was  a  cost  issue. 
He  indicated  that  the  type  of  maintenance  effort  that  is  established  and  contracted  for,  will 
probably  be  driven  by  cost.  [Ref.  13]  This  comment  seemed  to  fall  in  line  with  the  point 
raised  by  Mr.  McCaffrey  in  the  previous  Chapter  that  maintenance  costs  account  for  60- 
70%  of  the  total  software  resources.  [Ref.  9:p.  2] 

D.  SUMMARY 

This  Chapter  has  presented  an  overview  of  the  current  practices  and  applications, 
and  specific  contracting  related  issues  regarding  expert  system  development  and  acquisi¬ 
tion  in  the  public  sector.  In  presenting  the  expert  systems  development  and  acquisition 
issues  raised  by  representatives  from  industry,  it  is  evident  that  there  are  striking 
similarities  between  the  public  and  private  sectors.  As  alluded  to  earlier  in  this  Chapter, 
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one  reason  for  the  similarities  between  Government  and  industry  is  that  the  two  sectors 
often  combine  resources  through  conferences,  psqiers,  and  research.  While  not  addressing 
specific  expert  systems,  this  Chq>ter  focused  on  commonly  cited  problems  and  contracting 
issues  industry  professionals  deemed  significant  to  understanding  current  trends.  A 
combined  private  and  public  sector  discussion  will  be  presented  in  a  number  of 
conclusions,  derived  from  the  information  gathered  as  a  result  of  this  research,  in  Ch£q>ter 
V. 
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IV.  CONTRACT  TYPES  IDENTIFIED  AND  DISCUSSED 


A.  INTRODUCTION 

According  to  Major  Attanasio,  "the  single  most  important  document  in  the 
procurement  of  custom  designed  software  is  the  contract  itself."  [Ref.  2:p.  26]  Not  only 
does  the  contract  serve  as  a  basis  for  explicitly  stating  what  is  intended,  but  it  is  also  "one 
of  the  major  pricing  aids  available  to  a  buyer"  due  to  the  varied  contract  types  available. 
[Ref.  14;p.  181]  The  contract  is  essentially  a  collective  legal  agreement  between  the 
buyer  and  seller  (Government  and  industry),  clearly  delineating  the  responsibilities  for 
both  parties. 

Government  and  industiy  use  two  broad  contract  categories:  fixed-price,  and  cost- 
reimbursement  type  contracts.  The  Federal  Acquisition  Regulation  (FAR)  outlines  each 
contract  type,  as  well  as  the  various  factors  involved  in  selecting  and  applying  them. 
[Ref.  15:  Part  16]  The  specific  contract  types  contained  in  diese  nvo  categories  range 
from  firm-fixed-price,  to  a  cost-plus-fixed-fce.  The  fiim-fixed-price  type  contract  places 
full  responsibility  for  performance  costs  and  profit  on  the  contractor.  The  cost-plus-fixed- 
fee  type  contract  places  minimum  responsibility  on  the  contractor  for  performance  costs, 
and  the  ensuing  profit  fee  is  fixed.  The  contract  types,  as  listed  in  the  FAR,  are  identified 
in  the  following  sections. 

This  (Chapter  will  provide  a  discussion  of  the  two  categories  of  contract  types,  and 
how  each  contract  type  applies  to  software,  and  more  importantly,  expert  systems 
acquisition. 
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B.  FIXED-PRICE  TYPE  CONTRACTS 


According  to  the  FAR,  "fixed-price  types  of  contracts  provide  for  a  firm  price  or, 
in  appropriate  cases,  an  adjustable  price."  [Ref.  15:  16.201]  In  a  fixed-price  type  arrange¬ 
ment,  the  contractor  is  required  to  make  delivery  of  the  goods  and/or  services  for  a  given 
price,  on  time,  and  in  accordance  with  the  specifications  of  the  contract,  regardless  of 
costs  incurred.  In  doing  so,  the  Government  agrees  to  pay  the  contractor  a  fixed-price. 
The  following  discussion  addresses  four  general  categories  of  the  fixed-price  type 
contracts,  as  presented  by  Major  Attanasio,  as  they  jq>ply  to  software.  Their  application 
to  expert  systems  development  and  acquisition  will  be  presented  throughout  based  on 
input  firom  Mr.  Crossen  and  Mr.  Khan. 

1.  Firm-Fixed-Price  (FFP) 

As  previously  indicated,  the  FFP  contract  presents  the  least  amount  of  risk 
to  the  Government.  The  FFP  contract  "provides  for  a  price  that  is  not  subject  to  any 
adjustment  on  the  basis  of  the  contractor’s  cost  experience  in  performing  the  contract", 
and  imposes  "a  minimum  administrative  burden  upon  the  contracting  parties."  [Ref.  15: 
16.202-1]  According  to  Mr.  Crossen,  a  FFP  contract  is  a^ropriate  for  commercial  off- 
the-shelf  type  purchases,  but  not  for  expert  systems  in  the  design  or  development  stage. 
He  felt  that  a  FFP  or  even  a  FPIF  type  contract  would  be  appropriate  only  after  the 
operational  prototype  expert  system  was  developed,  and  the  process  was  beginning  to 
enter  the  deployment  phase  of  the  life  cycle  process.  [Ref.  11] 
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2.  Fixed-Price  with  Economic  Price  Adjustment  [FPE] 

The  FPE  type  contract  allows  for  upward  or  downward  price  adjustments 
to  the  contract  based  on  the  following  three  economic  factors: 

(1)  Established  Prices 

(2)  Actual  Costs  of  Labor  or  Material 

(3)  Cost  Indexes  of  Labor  or  Material 

In  allowing  for  such  adjustments,  the  Government  and  the  contractor  are  seeking  a  certain 
level  of  protection  from  these  economic  factors.  [Ref.  15:  16.203]  While  not  routinely 
used  for  software  acquisition  or  development,  it  is  possible  according  to  Major  Attanasio, 
and  therefore  should  be  included. 

3.  Fixed-Price-Incentive  (FPIS  or  FPIF) 

"A  fixed-price-incentive  contract  is  a  fixed  price  contract  that  provides  for 
adjusting  profit  and  establishing  the  final  contract  price  by  a  formula  based  on  the 
relationship  of  final  negotiated  total  cost  to  total  target  cost."  [Ref.  15:  16.204] 

FPI  type  contracts  are  used  as  a  means  to  "incentivize"  the  contractor  to 
control  costs.  If  the  final  cost  is  greater  than  the  target  cost,  then  the  contractor  loses 
profit.  If,  however,  he  manages  to  complete  the  contract  with  the  final  cost  being  less 
than  the  target  cost,  his  profit  will  be  greater.  As  indicated  previously,  Mr.  Crossen  felt 
that  a  FPI  type  contract  would  be  sqrpropriate  when  the  contractor  is  in  the  production 
phase,  and  sufficient  cost  or  pricing  data  were  not  available  to  negotiate  a  FFP.  [Ref.  1 1  ] 
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4. 


Firni'Fixed-Price,  Level  of  Effort  (FFP4^0E) 


A  FFPjLOE  type  contract  requires  the  contractor  to  perform  a  specified 
"level  of  effort",  for  a  specified  amount  of  time.  The  work  to  be  performed  is  stated  in 
general  terms  thereby  allowing  the  contractor  a  certain  degree  of  flexibility.  The 
Government  is  then  required  to  pay  a  fixed  dollar  amount  to  the  contractor  for  the  work 
performed.  [Ref.  15:  16.207-1]  Regarding  software  acquisition  and  development,  the 
FFP,LOE  is  "typically  used  for  feasibility  type  study  in  a  research  or  development  effort." 
[Ref.  2:p.  31]  In  the  expert  system  arena,  the  FFP,LOE  or  even  a  CPFF,LOE  was 
recommended  for  maintenance  efforts.  The  FFP,LOE  contract  was  recommended  by  Mr. 
Crossen  for  situations  where  the  "customer",  or  user,  would  have  to  "stay  out  of  the 
system."  The  CPFF,LOE  would  be  appropriate  for  situations  where  the  user  had  to  "get 
into  the  system."  [Ref.  11]  In  other  words,  when  the  user  can  actually  enter  the  system 
and  manipulate  data,  there  is  a  greater  potential  for  requiring  the  maintenance  technician 
to  be  involved  in  more  troubleshooting  than  he  normally  would  be.  When  the  user  has 
no  need  to  enter  and  manipulate  the  system  data,  there  is  less  potential  for  problems  to 
arise  as  a  result  of  outside  interaction. 

C.  COST  REIMBURSEMENT  TYPE  CONTRACTS 

Cost  type  contracts  are  best  suited  for  use  when  uncertainties  exist  in  being  able 
to  accurately  estimate  costs.  The  contractor  is  allowed  to  recover  payment  for  all  "allow¬ 
able  and  allocable"  costs.  The  FAR  describes  these  type  contracts  in  the  following 
manner: 

These  contracts  establish  an  estimate  of  total  cost  for  the  purpose 
of  obligating  funds  and  establishing  a  ceiling  that  the  contractor  may  not 
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exceed  (except  at  its  OAvn  risk)  without  the  approval  of  the  contracting 
officer.  [Ref.  15:  16.301-1] 

The  following  discussion  presents  the  various  cost  reimbursement  type  contracts, 
and  sqjplies  them  to  the  development  and  acquisition  of  expert  systems  software. 

1.  Cost-Sharing  (CS) 

Although  not  discussed  in  great  detail,  the  cost-sharing  type  contract  was 
mentioned  as  an  alternative.  In  this  type  contract,  the  contractor  receives  no  fee,  and  only 
gets  reimbursed  for  an  agreed  upon  portion  of  his  allowable  costs.  In  this  instance,  the 
contractor  generally  agrees  to  absorb  a  certain  portion  of  his  costs  in  expectation  of 
receiving  compensating  benefits  in  the  future.  [Ref.  15:  16.303]  While  not  widely  used 
in  expert  systems  development  and  acquisition,  the  possibility  exists  for  its  potential  use. 
A  firm  that  is  largely  in  the  research  and  development  business  could  view  this  type 
contract  as  a  stepping  stone  to  other  business  ventures.  [Ref.  6] 

2.  Cost-Plus-Incentive-Fee  (CPIF) 

The  CPIF  type  contract  is  a  cost-reimbursement  contract  that  provides  for 
an  initially  negotiated  fee  that  is  adjusted  based  on  the  total  allowable  costs  and  the  total 
target  costs.  Initially,  the  target  cost,  target  fee,  maximum  and  minimum  fee,  and  the 
adjustment  formula  are  generally  agreed  upon.  Impending  on  the  cost-share  ratio,  the  fee 
is  adjusted  upward  or  downward  depending  upon  whether  or  not  the  contractor  completes 
the  contract  below  or  above  the  target  cost.  The  key  is  to  reach  an  accord  with  the 
contractor  based  on  the  target  cost  and  fee  adjustment  formula  likely  to  motivate  him  to 
manage  effectively  and  efficiently.  The  CPIF  is  tqrpropriate  for  development  and  test  type 
programs.  [Ref.  15:  16.404-1] 
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3. 


Cost-Plus-Award-Fee  (CPAF) 


In  this  cost-reimbursement  type  contract,  the  contractor  is  allowed  an 
agreed  upon  base  amount  fixed  at  the  inception  of  the  contract.  In  addition  to  the  base, 
an  award  is  authorized  that  the  contractor  may  or  may  not  be  eligible  for.  The  eligibility 
is  based  on  the  Government’s  evaluation  of  the  contractor’s  performance  in  pre¬ 
determined  areas  such  as  quality,  timeliness,  technical  ingenuity,  and  cost-effective 
management.  The  general  criteria  for  the  contractor’s  perfonnance  are  stated  in  the  terms 
of  the  contract.  By  monetarily  rewarding  the  contractor  for  his  performance  in  given 
areas,  he  is  incentivized  to  maintain  a  reasonably  high  level  of  performance  standards  in 
order  to  gain  the  maximum  reward.  [Ref.  15:  16.404-2]  As  addressed  in  the  previous 
section,  both  the  CPIF  and  the  CPAF  type  contracts  are  ideally  suited  for  the  early  stages 
of  the  rapid  prototyping  process. 

4.  Cost-Plus-Fixed-Fee  (CPFF) 

A  CPFF  contract  allows  for  the  contractor  to  be  paid  a  fixed  fee  negotiated 
and  agreed  upon  at  contract  inception.  While  adjustments  can  be  made  based  on  changes 
in  the  work  to  be  performed,  the  fee  is  generally  fixed.  In  this  type  of  cost-reimburse¬ 
ment  arrangement,  the  contractor  is  allowed  the  fee  due  to  the  potential  risks  involved  in 
the  performance  of  work.  This  type  of  arrangement  does  not,  however,  provide  a  great 
deal  of  incentive  for  the  contractor  to  control  costs.  Under  the  CPFF  type  contract 
arrangement,  the  Government  assumes  the  majority  of  risk.  [Ref.  15:  16.306]  In  the 
expert  systems  arena,  the  CPFF  is  one  vehicle  recommended  for  use  in  the  research  and 
development  phase  of  the  DoD  life-cycle  acquisition  process.  [Ref.  6]  Because  of  the 
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greater  risk  assumed  on  the  part  of  the  contractor  in  this  phase,  die  CPFF  arrangement 
ensures  a  certain  fee  in  addition  to  the  allowable  and  allocable  costs.  [Ref.  11] 

D.  CHOOSING  THE  APPROPRIATE  ALTERNATIVE 

The  contract  types  previously  discussed  are  available  to  the  Government  and 
contractors  as  a  means  to  "provide  needed  flexibility  in  acquiring  the  large  variety  and 
volume  of  supplies  and  services  required  by  agencies."  [Ref.  15:  16.101]  The  selection 
of  the  proper  contract  arrangement  is  basically  a  matter  for  negotiation  and  the  application 
of  sound  judgement.  The  qiplication  of  sound  judgement  throughout  every  phase  of  the 
process  is  essential  to  a  successful  project.  The  key  of  contract  selection  is  to  negotiate 
a  contracting  arrangement,  and  either  price  or  estimated  cost  and  fee: 

...that  will  result  in  reasonable  contractor  risk  and  provide  the 
contractor  with  the  greatest  incentive  for  efficient  and  economical 
perfoimance.  [Ref.  15:  16.103] 

By  selecting  the  s^propriate  contracting  arrangement,  a  reasonable  level  or  share  of  the 
cost,  schedule,  technical,  and  support  risk  is  assigned  to  each  party.  The  ideal 
arrangement  is  one  that  has  the  Government  and  the  contractor  walking  away  feeling  like 
they  both  received  a  fair  and  reasonable  deal;  essentially  a  "win-win"  situation. 

Management  control  emerged  as  the  key  ingredient  throughout  the  entire 
development  and  acquisition  process.  Both  Government  and  industry  identified  the  need 
for  management  involvement  and  control  in  every  area  of  the  acquisition  process. 

The  industry  feeling  is  that  a  CPIF  or  a  CPAF  is  appropriate  for  the  concept, 
definition/design,  and  development  stages  of  the  DoD  life-cycle  management  process  for 
automated  infonnation  systems.  These  stages  of  the  rapid  prototyping  process  are  the 
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times  when  the  contractor  is  likely  to  incur  unanticipated  costs.  While  the  iterative 
process  entailed  in  n^id  prototyping  tends  to  significantly  reduce  overall  costs,  any 
developmental  stage  will  have  unforseen  expenses.  According  to  Mr.  Khan,  the  iterative 
process  tends  to  reduce  costs  by  foregoing  a  great  number  of  the  changes  and 
modifications  that  software  systems  developed  under  non-iterative  methodologies  tend  to 
experience.  For  this  reason,  and  to  reward  the  contractor  for  efficiency  and  effectiveness, 
the  CPIF  and  the  CPAF  are  recommended.  [Ref.  6] 

E.  SUMMARY 

By  presenting  the  various  contract  arrangements  outlined  in  the  FAR,  this  Chapter 
is  meant  to  provide  a  basic  understanding  of  die  primary  contract  types  available.  In 
iqiplying  the  contracting  arrangement  to  the  development  and  acquisition  of  expert 
systems,  the  popular  opinion  of  Government  and  industry  is  to  award  a  cost  type  contract 
in  the  early  stages  of  research,  concept  definition,  design,  and  development.  Whatever 
the  contracting  arrangement,  the  key  is  to  ensure  iq>propriate  allocation  of  risk,  a  quality 
product  or  service  for  the  Government,  and  a  fair  and  reasonable  profit  for  the  contractor. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

This  research  has  lead  to  specific  conclusions  regarding  the  development  and 
acquisition  of  expert  systems.  The  following  conclusions  will  be  addressed  individually 
and  then  followed  in  the  next  section  by  reconunendations. 

The  first  point,  and  possibly  the  most  significant,  is  the  question  of  what  an  expeit 
system  really  is.  Within  the  computer  software  community  there  exist  areas  of  specializa¬ 
tion.  One  such  area  is  artificial  intelligence  or  AI.  Expert  systems  fall  within  the 
purview  of  AI.  Within  the  AI  community,  there  are  those  who  believe  that  expert 
systems  ate  unique  in  and  of  themselves,  and  should  be  treated  differently  vis-a-vis 
typical  computer  software  applications.  The  research  indicated  that  expert  systems  should 
in  fact  be  treated  as  software,  and  that  similar  tqrproaches  be  taken  in  the  areas  of 
research,  design,  and  development.  The  development  and  acquisition  of  these  software 
systems  is  addressed  in  the  DoD  Directive  7920.1  regarding  life-cycle  management  of 
automated  information  systems.  [Ref.  5;p.  16]  Program  managers  and  contracting  officers 
have  to  remember  that  they  are  essentially  dealing  with  software  issues. 

Another  point  brought  about  as  a  result  of  this  research  is  the  methodology  used 
in  the  development  of  expert  systems.  The  traditional  or  "waterfall"  jqjproach  does  not 
fit  well  with  the  peculiarities  of  expert  systems  development.  It  requites  firm  specifica¬ 
tions  or  statements  of  work  early  on  in  the  development  process.  Because  of  the  iterative, 
or  growing  and  re-defining  type  of  developmental  iqrproach  in  expert  system  development, 
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the  waterfall  development  sq>proach  does  not  work  for  developing  expert  systems.  The 
need  for  up-front  specifications  restricts  the  developer  in  crafting  a  system  that  will 
properly  meet  the  needs  of  the  user.  DoD  documentation  standards  such  as  DoD  Standard 
7935.1,  Automated  Data  Systems  (ADS)  Documentation  Standards  (April  1984),  tend  to 
emphasize  hardware  vice  software  requirements.  Because  of  die  different  characteristics 
of  expert  systems  as  opposed  to  conventional  software,  i.e.,  expert  systems  attempt  to 
model  human  thought  processes,  and  the  emphasis  on  hardware,  the  developer  cannot 
effectively  document  the  expert  system  application  as  it  would  tqipear  to  the  user.  [Ref. 
4:p.  33]  The  waterfall  approach  essentially  removes  the  user  from  the  development 
process,  thereby  restricting  the  developer  from  being  able  to  meet  the  specific  user  needs. 
[Ref.  16:  88,102] 

Documentation  of  the  expert  system  development  process  is  important,  however, 
and  should  be  required  throughout  the  process.  A  recommended  sqiproach  is  that  of  a 
"Progress  Notebook"  that  would  entail  a  "historic  trace"  of  the  entire  expert  system 
development  process.  The  "Progress  Notebook"  would  tend  to  "fill  in  the  gjq>s  left  by  the 
standard  documents."  [Ref.  4:p.  38] 

The  popular  and  preferred  method  of  expert  system  development  is  known  as  the 
"rapid  prototyping"  approach.  This  method  allows  for  the  user  to  remain  closely  involved 
throughout  the  development  process.  The  contractor  develops  a  prototype  system  that  is 
tested  against  the  user’s  requirements.  Because  the  initial  system  is  defined  in  "broad 
functional  terms"  as  identified  in  Chapter  n,  the  contractor  is  afforded  greater  flexibility 
in  meeting  the  user’s  requirements.  [Ref.  2:p.  37]  Such  descriptions  could  be  in  terms 
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of  what  the  system  is  expected  to  do,  as  opposed  to  looks  and  language  lequiiements. 
By  describing  the  system  in  functional  as  opposed  to  design  type  terms,  the  developer  is 
afforded  the  latitude  to  adequately  respond  to  the  users  needs  throughout  the  iterative 
process.  If  described  in  specific  design  type  terminology,  the  developer  is  then  restricted 
in  developing  the  proposed  system.  Specific  issues  of  integration,  the  continuous  need 
for  user  and  domain  expert  system  participation,  and  the  need  for  dynamic  documentation 
throughout  the  process  are  easily  managed  in  the  context  of  the  rq^id  prototyping  model. 
[Ref.  5:p.  15] 

While  varying  somewhat  in  the  identification  of  specific  problem  areas,  industry 
and  Government  seem  to  be  in  general  agreement  regarding  questions  concerning  contract 
type,  cost,  schedule,  technical  performance,  and  maintenance.  Continued  Government  and 
industry  interaction  through  conferences,  publication  of  professional  pq)ers,  and 
communication,  will  ensure  these  concerns  and  problem  areas  will  continue  to  be 
addressed  and  potential  solutions  attained. 

B.  RECOMMENDATIONS 

As  a  result  of  this  research,  the  following  recommendations  are  made; 

1.  Continuous  Correlation  Between  Expert  Systems  and  Conventional 
Software 

Although  expert  systems  are  a  unique  q>plication  of  a  software  system, 
they  are  in  fact  software.  [Ref.  4:p.  33]  This  research  has  shown  that  conventional 
software,  and  expert  systems  development  and  acquisition  methodologies,  are  essentially 
the  same.  Issues  of  concern  in  the  software  development  field  are  virtually  mirrored  in 
the  expert  systems  arena.  Issues  such  as  requirements  changes,  specifications,  technical 
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perfonnance,  cost  and  management  control  ring  clear  in  both  fields.  As  indicated 
previously  in  Chjq>ter  IV,  the  question  of  software  maintenance  is  not  only  a  concern,  but 
significantly  impacts  life-cycle  cost  control  in  each  phase  of  the  process.  Because  of  the 
similarities  brought  forth  in  this  research  effort,  it  would  stand  to  reason  for  the  two  fields 
to  maintain  open  lines  of  communication.  Industry  and  Government  recognition  of  these 
issues  in  software  development  and  acquisition  projects  is  essential. 

2.  Use  of  the  Rapid  Prototyping  Methodology 

In  contracting  for  the  development  and  acquisition  of  expert  systems,  the 
iterative  rapid  prototyping  process  should  be  used.  Currently  the  preferred  method 
throughout  industry  and  the  Federal  Government,  the  rapid  prototyping  process  facilitates 
the  necessary  management  control  in  developing  and  acquiring  expert  systems.  Rq>id 
prototyping  additionally  ensures  that  the  needs  of  the  user  wiU  adequately  be  met.  A 
common  theme  throughout  this  research  has  been  the  link  between  the  various  problems 
encountered  in  developing  and  acquiring  an  expert  system,  and  the  process  used.  Both 
industry  and  Government  professionals  tend  to  agree  that  application  of  the  nq>id 
prototyping  method  can  significantly  reduce  the  problems  currently  encountered.  Con¬ 
cerns  with  requirements  changes,  specifications,  and  user-developer  interaction  are 
addressed  throughout  this  process.  The  rapid  prototyping  methodology  allows  for 
coordination  of  user  requirements  throughout  the  process.  Because  coordination, 
communication,  and  management  control  are  inherent  to  the  rapid  prototyping  process, 
the  Government  and  contractor  reap  the  benefits  of  schedule  and  cost  control  measures. 
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3.  Education  and  Training  of  Users,  Contracting  Officers,  and  Program 
Managers 

Viewing  expert  systems  from  a  sofrware  perspective,  and  adequately 
applying  the  management  techniques  involved  in  the  rapid  prototyping  methodology,  can 
only  be  brought  about  through  education  and  training.  From  a  managerial  viewpoint, 
knowledge  and  understanding  of  the  proposed  system  is  just  a  small  part  of  the  overall 
understanding  required.  In  order  to  properly  apply  judgement,  and  make  informed 
decisions  about  the  program  or  acquisition,  the  manager  and  contracting  officer  have  to 
understand  the  industry  as  well  as  the  proposed  system.  From  a  user  and  developer 
perspective,  a  thorough  knowledge  and  understanding  of  the  system,  system  operating 
envirorunent,  applications,  and  alternatives  will  greatly  benefit  the  overall  process. 
Through  educational  courses  and  professional  conferences  and  journals,  people  involved 
at  each  level  of  the  development  and  acquisition  process  will  be  better  able  to  respond 
in  this  dynamic  environment. 

4.  Continuous  Revision  and  Updating  of  Government  Regulations  and 
Specifications 

Current  trends  in  both  the  Federal  Government  and  industry  reflect  concern 
for  revising  and  updating  acquisition  regulations  and  Government  specifications.  In 
offering  the  views  of  the  Aerospace  Industries  Association  (AIA)  regarding  the  Secretary 
of  Defense’s  "Defense  Management  Report"  of  July  1989,  Mr.  Don  Fuqua,  President  of 
AIA,  recommended  that  "industry  should  be  required  to  participate  in  the  initial 
development  of  future  regulatory  changes."  [Ref.  17:pp.  1-1, 2-1 1]  This  recommendation 
falls  directly  in  line  with  the  sentiment  from  both  industry  and  Government  interviewees; 
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that  not  only  should  acquisition  streamlining  continue,  but  the  Government  and  industry 
should  work  together.  By  working  together,  issues  regarding  the  abundance  of  regulations 
and  specifications,  and  contracting  arrangements  best  suited  for  the  situation  can  be 
addressed  and  responded  to.  As  indicated  previously  in  this  papct,  communication 
between  Govenunent  and  industry  is  not  just  a  good  idea,  but  essential  to  the  effective 
and  efficient  a^lication  of  the  development  and  acquisition  process. 

C.  ANSWERS  TO  RESEARCH  QUESTIONS 

1.  What  unique  contractual  considerations  are  involved  in  procuring  an 

expert  system  for  use  in  the  DoD? 

As  indicated  throughout  this  research,  there  are  few  "unique"  considerations 
regarding  the  acquisition  of  expert  systems.  On  the  contrary,  it  is  important  to  note  that 
expert  systems  should  be  viewed  in  the  purview  of  software  development  and  acquisition. 
The  primary  difference  between  expert  systems  and  conventional  software,  .  s  discussed 
previously,  is  that  expert  systems  "manipulate  knowledge"  vice  "data."  [Ref.  l:p.  24] 
Expert  systems  should  be  acquired  based  on  the  factors  of  need,  scope,  application,  and 
estimated  cost  of  the  specific  project  at  hand.  Questions  regarding  whether  to  purchase 
a  commercial  shell,  or  develop  the  system  from  scratch,  are  a  function  of  these  factors. 
While  the  results  of  this  research  stress  the  use  of  rapid  prototyping  for  expert  system 
development  and  acquisition  as  opposed  to  the  waterfall  method,  further  research  could 
be  conducted  to  determine  other  feasible  variations  or  alternatives.  Other  concerns  and 
problems  identifted  as  a  result  of  this  research  virtually  mirrored  those  brought  up  in 
previous  research  in  software  development. 
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2.  Of  the  expert  systems  currently  acquired  by  the  DoD,  how  satisfied 
are  the  agencies  with  these  systems? 

The  people  interviewed  throughout  this  research  effort  indicated  satisfaction 
with  the  different  expert  systems  at  their  disposal.  Because  of  the  abundance  of  informa¬ 
tion  regarding  other  significant  problems  in  the  expert  system  development  and 
acquisition  arena,  the  actual  systems  themselves  were  not  addressed.  Tlie  interviewees 
reflected  satisfaction  with  the  actual  systems,  however,  they  were  specific  in  identifying 
problem  areas  as  were  discussed  in  Chapters  n  and  m. 

3.  What  were  the  contractual  problems  associated  with  the  acquisition  of 
these  systems? 

From  the  Government  perspective,  the  five  most  commonly  identified 
problem  areas  in  expert  systems  developnwnt  and  acquisition  were  identified  in  Table  2-2. 
The  primary  recommendations  to  remedy  Aese  problems,  or  at  least  lessen  their  impact 
are: 

*  Use  of  rapid  prototyping  development  and  acquisition  methodology 
Training  qualified  and  experienced  personnel 
By  applying  the  rapid  prototyping  methodology  to  the  development  and  acquisition  of 
expert  systems,  user  and  developer  interaction  can  be  managed.  By  ensuring  early  and 
continuous  user  interaction  throughout  the  process,  requirements  can  be  responded  to,  and 
adapted,  thereby  limiting  the  level  of  confusion  over  what  the  user  actually  wants  and 
needs. 

As  addressed  in  the  recommendations  section,  education  and  training  of  the 
personnel  involved  in  the  development  and  acquisition  of  expert  systems  is  essential. 
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Improved  management,  communication,  and  coordination  can  be  assured  through  ensuring 
that  these  people  have  an  adequate  understanding  of  the  system,  system  environment, 
acquisition  methodology,  and  available  alternatives. 

4.  What  are  the  observations  of  industry  (both  user  and  producer) 
regarding  satisfaction  in  the  application  and  performance  of  expert 
systems,  and  any  special  considerations  involved  in  the  acquisition  of 
these  systems? 

While  the  problem  areas  identified  by  industry  varied  somewhat  from  diose 
of  the  Government,  their  major  concerns  were  essentially  the  same.  Additionally,  industry 
expressed  satisfaction  with  the  general  application  of  currendy  used  expert  systems.  The 
predominant  concerns  of  industry  regarding  expert  systems  development  and  acquisition 
were  presented  in  Table  3-1. 

Industry  sentiment  seems  to  be  in  line  with  the  Government’s  in  that 
unclear  requirements,  as  well  as  requirements  changes,  often  lead  to  cost  and  schedule 
overruns.  Industry  representatives,  like  their  Government  colleagues,  felt  that  both 
education  and  use  of  the  rsqiid  prototyping  development  and  acquisition  methodology 
would  significantly  enhance  the  overall  process. 

In  discussing  the  issues  of  Government  regulations  and  specifications,  the 
words  "burdensome",  "confusing",  "over-abundance",  and  "ambiguous"  often  came  up. 
Industry  professionals  agreed  that  the  Government  needed  to  pay  particular  attention  to 
regulations  and  specifications.  By  continually  reviewing  these,  and  even  including 
industry  in  the  process,  the  overall  system  could  incur  substantial  savings  in  the  areas  of 
cost,  technical  performance,  schedule,  and  maintenance  support. 
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5. 


What  are  the  specific  issues  regarding  contract  type,  contract 
administration,  research  and  development  (R&D),  prototype  develop¬ 
ment,  production,  and  maintenance  of  expert  systems? 

The  primary  contracting  issues  were  broken  down  into  the  areas  of  cost, 

schedule,  technical  performance,  maintenance  support,  and  contract  type.  Contract 

administration,  while  being  an  area  of  concern,  was  too  broad  in  scope  to  include  in  this 

research  effort.  For  this  reason,  contract  administration  issues  such  as  warranties,  contract 

changes,  contract  inteipretation,  disputes,  and  legal  implications,  are  reconunended  for 

future  research  efforts. 

Each  of  the  areas  identified  above  is  linked  to  management  and 
methodology.  The  methodology  issue  is  continually  referred  to  as  being  the  key  to  a 
program  manager’s  or  contracting  officer’s  ability  to  adequately  handle  the  situation. 
Proper  planning,  and  jqjpropriate  acknowledgement  of  the  contracting  issues  identified 
above,  throughout  every  phase  of  the  development  and  acquisition  process,  is  one  means 
to  reasonably  ensure  success. 

D.  RECOMMENDATIONS  FOR  FUTURE  STUDY 
1.  Contract  Administration  Issues 

Examine  pertinent  contract  administration  issues  such  as  warranties, 
disputes,  contract  inteq7retation,  contract  changes,  adjustments,  and  legal  questions  from 
an  expert  system  acquisition  perspective.  As  previously  indicated,  contract  administration 
is  definitely  an  area  of  concern  in  the  overall  development  and  acquisition  process.  A 
review  of  the  various  studies,  and  literature  on  these  topics  for  conventional  software 
acquisition  can  be  directly  related  to  expert  systems.  An  interesting  question  is  that  of 
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liability  and  risk  associated  widi  an  expert  system,  i.e.,  responsibility  for  inaccurate 
decisions  or  results  based  on  the  information  provided  by  an  expert  system.  Issues  of 
legal  responsibility  may  or  may  not  be  addressed  in  the  warranty.  If  addressed,  then  how 
are  the  issues  enforced?  The  answer  may  lie  in  the  realm  of  contract  interpretation,  or 
clearly  delineating  within  the  context  of  the  a{^ropriate  contract  clauses  what  is  intended 
by  the  Government  and  the  contractor. 

2.  Formalized  Education  and  Training  in  Software  Development  and  Ac< 
quisition 

Examine  the  current  system  of  education  and  training  in  the  area  of 
software  development  and  acquisition.  Compare  the  current  system  with  various 
alternative  approaches  such  as  formal  training  for  procurement  personnel  regarding 
software  and  hardware  terminology  and  £q}plications.  The  focus  of  such  training  would 
be  in  the  area  of  expert  system  design,  development,  and  production  techniques  as  they 
tqjply  to  expert  systems.  Alternatives  can  be  obtained  through  interviews  with  Govern¬ 
ment  and  industry  professionals.  As  identified  throughout  diis  research  effort,  the  need 
for  formal  education  and  training  at  every  level  of  the  software  development  and 
acquisition  process  is  essential. 

3.  Review  Development  and  Acquisition  Methodologies  for  Specific 
Expert  Systems  Applications 

Analyze  and  compare  various  expert  systems  jqjplications  from  a 
development  and  acquisition  perspective.  Through  in-depth  analysis  of  selected  expert 
systems  projects,  an  accurate  assessment  of  the  development  methodologies  used  can  then 
be  made.  By  studying  actual  expert  systems  projects,  the  various  methodologies  can  be 
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assessed  as  to  their  effectiveness  and  efficiency.  This  type  of  research  effort  would 
provide  professionals  in  the  field  of  expert  systems  development  and  acquisition  with  the 
appropriate  understanding  of  the  best  suited  methodology  to  apply. 

4.  Review  of  Government  Regulations  and  Specifications 

Examine  current  Government  regulations,  specifications,  and  standards  for 
appropriate  application  to  expert  systems  development  and  acquisition.  By  identifying 
those  Government  regulations,  specifications,  and  standards  that  are  ambiguous  or 
repetitive,  measures  can  then  be  made  to  clarify,  update,  or  possibly  delete  applicable 
items.  A  joint  project  including  Government  and  industry  representatives  could  be 
initiated  as  a  means  to  potentially  identify  possible  issues  of  concern.  Government 
standards  such  as  those  identified  in  Chapter  HI  may  be  able  to  be  condensed,  combined, 
updated,  or  possibly  even  deleted.  By  working  with  industry  and  Government  on  this 
project,  the  concerns  regarding  redundancy  and  overleq)  can  be  addressed. 

E.  SUMMARY 

As  a  result  of  this  research,  certain  key  areas  of  concern  from  the  Government  and 
industry  perspective  were  brought  to  light.  Education  and  training,  redundant  Government 
regulations,  specifications,  and  standards,  and  development  methodology,  were  frequently 
discussed.  Both  industry  and  Government  personnel  tended  to  agree  that  these  areas  must 
be  addressed  from  user  to  program  manager  to  contractor. 

Understanding  that  expert  systems  are  software,  and  that  they  should  be  developed 
and  acquired  in  a  similar  fashion  is  a  paramount  concern.  By  understanding  this 
correlation,  use  of  the  rapid  prototyping  methodology  arises  as  the  natural  vehicle  for 


50 


acquiring  an  expert  system.  Rather  than  change  the  Government  approach  to  software 
development,  a  proposed  method  to  work  within  this  system  has  been  recommended  in 
Table  2-5.  This  method  allows  for  the  iterative  prototype  development  process  required 
for  expert  systems,  as  well  as  the  DoD  requirement  to  follow  the  AIS  development 
process  via  the  milestone  approach.  By  "mapping"  the  two  processes  together, 
efficiencies  in  time,  schedule,  and  cost  can  be  attained  by  ensuring  an  adequate  level  of 
user-developer  interface  throughout. 

The  remaining  two  areas  of  concern  were  contract  type  and  maintenance.  Most 
interviewees  agreed,  as  does  the  author,  that  a  cost  type  contract  for  the  early  stages  of 
expert  system  design  and  development  is  advantageous  for  industry  as  well  as  the 
Government.  In  this  type  arrangement,  industry  is  rewarded  for  assuming  a  greater 
portion  of  the  cost  risk  associated  with  early  development  phases.  In  a  similar  vein,  as 
the  development  process  leads  to  an  operational  prototype,  and  enters  the  deployment 
phase  of  production,  the  level  of  risk  changes.  Because  of  the  shift  in  risk,  the  type  of 
contract  should  also  shift  to  reflect  the  change  by  assuming  a  fixed-price  type  of 
contracting  arrangement.  Whether  by  agreeing  to  one  contract  that  will  change  according 
the  stage  of  development,  or  by  contracting  for  each  phase  of  development  separately, 
these  types  of  contracting  arrangements  should  be  pursued.  The  decision  to  go  with  one 
contract  or  a  series  of  contracts  depends  primarily  on  time,  cost,  and  competition. 

While  being  an  important  issue  of  concern,  the  area  of  maintenance  is  often 
overlooked.  Because  of  the  large  proportion  of  cost  involved  in  maintenance  throughout 
the  life-cycle  of  the  system,  maintenance  should  be  required  to  be  addressed  and  planned 
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for  from  the  earliest  stages  of  concept  design  and  development.  By  starting  early  on  in 
the  rapid  prototyping  process,  the  user  and  developer  can  begin  to  respond  to  the 
questions  of  contractor  maintenance,  in-house  maintenance,  or  a  combination  of  the  two. 
The  need  for  maintenance  may  require  that  not  only  a  clause  be  added  to  the  contract 
addressing  this  issue,  but  that  the  proposed  "mapping"  of  the  DoD  AIS  model  and  the  ES 
life-cycle  model  be  changed  to  reflect  this  need  throughout  each  stage  of  development. 

Finally,  as  discussed  in  the  previous  section,  it  is  important  to  continue  addressing 
the  areas  of  contract  administration,  education  and  training,  development  and  acquisition 
methodologies,  and  Government  regulations,  speciJhcations,  and  standards.  The  rapidly 
changing  and  growing  field  of  artificial  intelligence  will  continue  to  raise  new  concerns 
over  these  areas. 
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APPENDIX  A 


LIST  OF  CONFERENCES  ATTENDED 


1.  Forum  on  Artificial  Intelligence  in  Acquisition  Management.  Sponsored  by  the 
Defense  Systenis  Management  College,  Fort  Belvior,  VA.  Hosted  by  the  Naval 
Postgraduate  School,  Monterey,  CA,  14-17  May  1990. 

2.  IEEE  Managing  Expert  System  Programs  and  Projects  Conference.  Sponsored 
by  the  IEEE  Computer  Society  Technical  Committee  on  Expert  Systems 
Applications.  Hosted  by  the  Holiday  Inn,  Bethesda,  MD,  10-12  September  1990. 
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APPENDIX  B 


LIST  OF  PEOPLE  INTERVIEWED 


1 .  Bulks,  Shield.  Artificial  Intelligence  Program  Analyst,  ALD/m,  Wright  Patterson 
Air  Force  Base,  OH,  Interview,  May  1990. 

2.  Conner,  Dave.  Procurement,  Intellicorp,  Washington,  D.C.,  Interview,  September 
1990. 

3.  Crossen,  Jim.  TRW,  Sunnyvale,  CA,  Interview,  September  1990. 

4.  Davis,  Laura.  Information  Technology  Division,  Naval  Research  Laboratories, 
Washington,  D.C.,  Interview,  September  1990. 

5.  Feinstein,  Jerald.  Mitre  Corporation,  McLean,  VA,  Interview,  September  1990. 

6.  Griesser,  Jay.  Technical  Application  Branch,  Navy  Finance  Center,  Qeveland,  OH, 
Interview,  August  1990. 

7.  Hoffman,  Ed.  Telesales  Representative,  Intellicorp,  Mountain  View,  CA, 
Interview,  September  1990. 

8.  Khan,  A.  F.  Umar.  7th  Communication  Group,  Program  Analysis  Division,  United 
States  Air  Force,  The  Pentagon,  Washington,  D.C.,  Interview,  August  -  October 
1990. 

9.  Kim,  Hun.  Artificial  Intelligence  Program  Manager,  Naval  Supply  Systems 
Command,  Washington,  D.C.,  Interview,  August  1990. 

10.  Klahr,  Phil.  Inference  Corporation,  El  Segiuido,  CA,  Interview,  October  1990. 

11.  Siegel,  Harry.  JAYCOR,  Vienna,  VA,  Interview,  October  1990. 
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